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ABSTRACT 

This final report discusses the activities and outcomes of a 
project that provided training on the Paraeducator Supervision Academy (PSA) 
Model (curriculum, instructional materials, and supervision of 
paraprofessionals ) to faculty who will prepare future school professionals. 
Training was provided to faculty and preservice students in schools and 
colleges of education, faculty in related services programs, staff developers 
and in-service school professionals, and parents in 32 replication sites in 
21 states and in the Department of Defense schools. Each demonstration 
consisted of two full days of training of potential trainers, as well as 
teachers and related services personnel. It addressed coordination among 
service providers by demonstrating the use of the package to multi- 
disciplinary teams. A total of 1,137 people participated in the training. 

Pre- and post-test self-perceived skills change data show significant 
differences in skill level. Interviews with 22 participants found that 77% of 
participants were using the concepts, activities, or materials presented in 
the demonstrations in some way. A 20-minute videotape was also developed 
during the course of the study discussing building better working 
relationships with paraprof essional school personnel and focusing on the work 
of three teachers at different grade levels and the students and 
paraeducators associated with them. (CR) 
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EXECUTIVE SUMMARY 

Paraeducator Supervision Academy (PSA) 

An Outreach Project Proposal (CFDA 84.324R) 

The PSA Outreach project provided training to faculty and preservice students in schools 
and colleges of education, faculty in related services programs, staff developers and inservice 
school professionals, and parents in 32 replication sites. The primary goal of the training was 
to provide the PSA Model, curriculum, instructional materials, and background knowledge 
regarding the supervision of paraprofessionals to faculty who will prepare future school 
professionals. The delivery model relied on the assembly of a multi-disciplinary audience, 
including parents of children with disabilities, at the replication site. 

The 1997 amendments to IDEA permit paraeducators to assist in the delivery of services 
to children with disabilities provided the paraeducator is trained and appropriately 
supervised. Unfortunately, school professionals tend to have little preparation to direct the 
work of paraeducators and most currently employed paraeducators perform their work with 
little accountability. There is great concern that poorly supervised and inadequately trained 
paraeducators may undermine the intended outcomes of students’ lEPs. 

Part of the problem is that few faculty members in Schools or Departments of Education 
in Institutions of Higher Education (IHE) have first-hand knowledge of paraeducator 
responsibilities, and fewer have texts or materials to provide appropriate information to 
future teachers and related service providers. So, new teachers are no better prepared than 
seasoned teachers. Moreover, few staff developers in LEAs are prepared to provide inservice 
training to teachers who arrive unprepared at the district’s doorstep. This project provided a 
training model, a set of materials, and technical assistance to implement training at multiple 
levels. 

Project Goals: 

1 . To demonstrate the Paraeducator Supervision Academy model at 32 replication sites. 

2. To provide ongoing technical assistance to IHE faculty and LEA staff developers. 
Importance: 

This project disseminated the PSA Model for preparation of school professionals who 
supervise paraeducators. It addresses the coordination among service providers by 
demonstrating the use of the package to multi-disciplinary teams including teacher 
preparation and related services faculty in IHEs as well as LEA staff developers, school 
professionals and parents. 
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PROJECT BRIEF DESCRIPTION 

The PSA Outreach project was developed to promote preparation of preservice and 
inservice teachers in the skills necessary for them to work effectively with paraeducators. 

The primary audience was faculty in schools and colleges of education and district-based 
staff developers and administrators. The secondary audience included inservice school 
professionals (teachers, administrators, and related services professionals) in 32 replication 
sites. 

The primary goal of the training was to provide the Paraeducator Supervision Academy 
(PSA) Model, curriculum, instructional materials, and background knowledge to faculty who 
will prepare future and current professionals in the supervision of paraeducators. The 
delivery model relies on the assembly of a multi-disciplinary audience at the replication site. 

The background and rationale for the project built on the 1997 amendments to IDEA that 
specifically permitted paraeducators to assist in the delivery of services to children with 
disabilities provided the paraeducator was trained and appropriately supervised. 
Unfortunately, school professionals have had no preparation in directing the work of 
paraeducators and most currently employed paraeducators perform their work with little 
direction and little accountability. There is great concern that poorly supervised and 
inadequately trained paraeducators may undermine the intended outcomes of students’ lEPs. 

Moreover, few school of education faculty members in Institutions of Higher 
Education (IHE) have first-hand knowledge of paraeducator responsibilities, and fewer have 
texts or materials to provide appropriate information to future teachers and related service 
providers. While generally it is true that staff developers in LEAs are better prepared to 
provide state-of-the-art training to inservice professionals, in this case they lack materials, 
resources and knowledge to fill the gap. 

This project provided a training model as well as technical assistance in the 
implementation of the model. It addressed coordination among service providers by 
demonstrating the use of the package to multi-disciplinary teams including teacher 
preparation and related services faculty in IHEs as well as LEA staff developers, school 
professionals and parents. 
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REPORT OF PROJECT ACTIVITIES AND OUTCOMES 

The original PSA Outreach action plan consisted of two major goals, and numerous 
objectives and activities were associated with each goal. This section of the report is 
organized in outline form with narrative information within each section. 

Goal 1. Demonstrate the PSA Model at 32 replication sites in 3 years. 

Objective 1.1. Put finishing touches on the PSA Model Package 

Activity 1.1.1. Research and develop reading materials packet 
The readings we initially assembled for the audiences consisted of the following 
citations. We determined the selections based on the backgroimd knowledge we believed that 
potential trainers would need to provide teachers with preparation on paraeducator 
supervision. At that point, there were few articles appropriate for these purposes. 

Council for Exceptional Children. (1998). The knowledge and skill standards for 
beginning paraeducators of children with exceptionalities. In CEC (Ed.). What every special 
educator must know: The international standards for the preparation and licensme of special 
educators, 3^^* edition . 

French, N.K. &. Pickett, A.L. (1997). Paraprofessionals in Special Education: Issues for 
Teacher Educators. Teacher Education and Special Education, 20{\) 61-73. 

French, N. K. (1996). A case study of a speech-language pathologist's supervision of 
assistants in a school setting: Tracy's story. Journal for Children's Communication 
Development, 7^(1), 103-110. 

French, N. K. (1998). Working together: Resource teachers and paraeducators. 

Remedial and Special Education, 19(6) 357-368. 

National Joint Committee on Learning Disabilities. (1999). Learning disabilities: Use of 
paraprofessionals. Learning Disability Quarterly, 22, 23-30. 

Rueda, R., & DeNeve, C. (1999). How Paraeducators Build Cultoal Bridges in Diverse 
Classrooms. Community Circle of Caring Journal, 3(2), 53-55 [Online]. Retrieved from: 
http://www.usc.edu/dept/education/CMMR/paraed/Rueda DeNeve article.html 
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Over the project period we added articles written by the principle investigator. These 
articles were included as examples of handouts trainers could use with their students. 

French, N. K. (2000). Taking time to save time: Delegating to paraeducators. Teaching 
Exceptional Children, 32(3), 79-83. 

French, N. K. (1999). Paraeducators: Who are they and what do they do? Teaching 
Exceptional Children, 32(1), 65-69. 

French, N. K. (1999). Paraeducators and teachers: Shifting roles. Teaching Exceptional 
Children, 32(2), 69-73. 

Up to 10 copies of the textbook, entitled Supervising Paraeducators in School 
Settings: A Team Approach, edited by Pickett and Gerlach, was also supplied to key 
participants at each demonstration site. It included important information about roles and 
responsibilities, ethical standards, teamwork, and the managerial aspects of paraeducator 
supervision. Although French, the principle investigator, authored the longest chapter in the 
text, she received no royalties or other financial benefits from the purchase of the text. 

Activity 1.1.2. Locate standards, legislation, guidelines for paraeducator employment 

This activity involved reviewing state websites, contacting state education agency 
persoimel, and assembling relevant legislation, policies, practices, handbooks or other 
relevant information for each state. Using the resulting database, we provided customized 
training materials to the participants in each state in which we held a demonstration. 

Activity 1.1.3. Negotiate video production work 

We sought bids from various companies regarding the production of the video. In the 
end, we returned to the university media center and determined that the facilities on campus 
were more affordable, and Just as professional as those found elsewhere. We negotiated 
price, filming quality, editing styles, and various other production details. Our total costs for 
the video remained within the budgeted amount. 

Activity 1.1.4. Create story board for video 
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In the first continuation report, we noted that this aspect was behind schedule because 
of the release of an unsatisfactory employee. We managed to bring the project up to schedule 

within SIX months. We wrote the script, created the story board, and established a filming 
schedule. 

Activity 1.1.5. Identify school professionals to participate in filming 

At each site, we collected permission forms from the parents of every child included 

m the video, as well as the teachers and paraeducators. None of the featured “talent” was paid 
for their time. 

Activity 1.1.6. Film video segments and on-site activities 

We filmed in three different schools, using two TV quality video cameras, shooting 
about 24 hours worth of both “A” and “B” footage. We focused on the work of three teachers 
at different grade levels and the students and paraeducators associated with them. 

Activity 1.1.7. Final editing of video segments 

We hired a professional voice actor to read the script which we then edited into the 
footage and the audio sequences recorded in the schools. The final editing required numerous 
steps and many reviews of draft products. In the end, we had a 19 minutes video that 
summanzed all aspects of the PSA demonstration. We provided multiple copies of the video 
to each demonstration site, as they requested. This ranged from as many 60 copies in 
Alabama, to 28 in Iowa, and as few as one or two in the remaining sites. 

Activity 1.1.8. Create for ms to evaluate the demonstration of PSA 

We used pre and post tests that had been developed for our use in Colorado to assess 
participants’ perceptions of their own knowledge and skills before and after training during 
the first 2 years of the project, but switched to a new single instrument that we developed 
after momtonng the results of these pre and post assessments for 1 8 months. The results of 
this aspect of evaluation will be discussed fiirther in the evaluation section of this report. 

Objective 1.2. Select 10-11 sites per year (32 sites in three years). 

Activity 1.2.1. Advertise through n ational journals, newsletters, the web page 
Although first year sites had already been selected and key contacts had written 
letters of support prior to the start of the project, the actual negotiations for demonstrations 
began m earnest only after the award letter was received by the university and that award 



ERIC 



8 



PSA Outreach Final Report 



.Page 8 






letter was dated September 1 8, 1998 — well after the school-year schedule had been 
established. Thus the first demonstration was delayed until the following summer. 

Activity 1.2.2. Submit proposals to present at conferences 

While waiting to present the first demonstration, the PI began in earnest to solicit 
agreement from other host sites. During the first project year, the PI submitted 8 proposals to 
present at national and regional conferences, made numerous phone calls to colleagues (e.g. 
IHE faculty members in schools of education or related services programs, CSPD 
coordinators in SEAs, Part C coordinators in lead agencies) across the country to generate 
interest. 

Activity 1.2.3. Develop and disseminate memos, fivers 

We developed and disseminated memos to every SEA, and made follow up phone 
calls to those that responded. We developed a flyer and a tri-fold brochure that explained the 
benefits of hosting a demonstration. The PI created a PowerPoint presentation that outlined 
the content of the PSA demonstration, the rationale and research behind the content, and 
explained the benefits and costs of hosting a demonstration. The PowerPoint presentation 
was used at 8 subsequent conferences. 

Activity 1.2.4. Advertise in pre-conference materials . 

We were only able to add flyers to folders and pre-conference materials for three of 
the conferences. 

Objective 1.3. Demonstrate the PSA Model. 

Activity 1.3.1. Meet with stakeholders to discuss implementation of the PSA Model . 

This activity was the most time consuming of all. Because the concept was new to many 
people, the telephone and face-to-face conversations took much longer than anticipated. The 
PI spent anywhere from 12-18 hours of phone and face to face conversations and sent 
numerous lengthy email messages to establish each of the initial site demonstrations. Some 
site demonstrations never materialized in spite of long hours of conversations, email 
conversations, letters and transfer of materials. In each of those cases, the contact person 
failed to follow through on arranging for the session. In some cases, the lack of follow- 
through was related to a lack of support from other key stakeholders in the state. However, 
those who heard about the demonstrations via conferences and heard the PowerPoint 
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presentation required much less time, many of their questions having been answered via 
PowerPoint presentation and face to face conversations at conferences. 

Activity 1.3.2. Demonstrate the PSA Model to the assembled audience 
We provided 32 demonstrations at sites in 21 states and in the Department of Defense 
schools (at their expense rather than the expense of the project). Each demonstration 
consisted of two full days of training of potential trainers (including xmiversity faculty, staff 
developers from schools districts, community college personnel, SEA representatives, and 
school administrators) as well as teachers and related services providers. We deliberately 
established audiences that consisted of participants with various roles in education. We 
wanted the potential trainers to see and hear for themselves the types of concerns and issues 
that teachers face every day and the lack of supports and supportive policies that they 
confront in their daily work with paraeducators. We also wanted them to hear, first hand, the 
answers available to teachers and related services providers and how the materials could be 
used to respond to the real problems of practice. 

The Participants 
Position Titles 

A total of 1 137 people participated in the 32 PSA trainings. Participant titles included 
‘professor’ (29), ‘administrator’ including directors of special education, principals, assistant 
principals (189), ‘consultant’ (9), ‘staff development coordinator’, ‘related services’ (47) 
‘teacher’ (526), ‘coxmselor’ (3) ‘paraeducator’ (88), ‘state department of education’ (30), 
‘parent’ (2), and ‘paraeducator’ (88). Many participants failed to list their job title and those 
are unknown. 

Ethnicity of Participants 

Table 1 shows the known participant ethnicities. The majority of participants were 
Caucasian, however a large number (425) of participants chose not to disclose their ethnicity. 
The recollection of trainers is that the number of African-American and Asian participants 
would be significantly higher if all participants had provided ethnicity information. 

Table 1 

Participant Ethnicity 

er|c 
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African-Am 


Asian 


Hispanic 


Native 

American 


Caucasian 


Blank or 
Multi Missing 


60 


6 


34 


7 


603 


2 425 



Activity 1.3.3. Discuss incorporating PSA into existing IHE curriculum and programs 
We held evening and lunch-time conversations with potential trainers at each 
demonstration site to consider ways to incorporate the materials into existing curriculum at 
IHEs and into staff development programs in LEAs. Potential trainers were given the email 
addresses and other contact information for project staff. 

Activity 1.3.4. Make mini-grants available to defray demonstration hosting costs 
Mini-grants were available to all demonstration sites. These were advertised in the 
PowerPoint presentation at conferences and were mentioned diuing numerous conversations 
with possible hosts. However, only 7 host sites requested mini-grant funds, leaving unused 
funds in that budget line to be moved into salary so that we could complete the remaining 
project activities during a fourth-year, no-cost time extension. Diuing the fourth year, we 
completed the remaining demonstrations and provided significant amounts of consultations 
and ongoing support to site hosts in promoting the incorporation of the content into teacher 
preparation efforts in the state. 

Goal 2. Provide ongoing technical assistance to IHE faculty and LEA staff developers. 

Objective 2.1. Provide consultation and technical assistance via email and phone 
We provided extensive follow up via email and telephone. A thematic analysis of email and 
phone conversation notes is included in the section on Evaluation Goal 2. 

Activity 2.1.1. Provide timely feedback based on pre and post test data to participants 
Although we have now analyzed all of the pre and post self-assessment data, we did not 
share it with participants. As we prepared to do so, we realized that this was not of significant 
interest to participants who were more concerned about how to use the materials and 
incorporate the concepts into practice. 

Activity 2.1.2. Provide participants with revisions and material updates. 

We used email to provide participants with revisions and updates. Even now, four and a half 
years after the start of the project, we get requests from participants to update them on new 
O 
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events and materials for training purposes. For example, the following message arrived on 
5 / 22 / 03 . 

Dear Dr. French: 

I attended a workshop you led for the Kaiserslautern School District back in 
SY 1999-2000 regarding working with paraprofessionals. I have since moved to the 
Virginia Beach City Public Schools division, and will be co-leading 2 division staff 
development workshops about issues involving paras (sic) in special education 
classrooms. Would it be possible for you to send me some of the information from 
the workshop? Or, if you have anything more current, or different, that you think 
might be helpful, that would be fine, too. Thanks so much for your time and 
attention. 

Sincerely, Anne S... 

Activity 2.1.3. Update all participants via the PSA Web Page 
The PSA webpage contained all of the worksheets and copies of the other materials that 
participants received, so that they could access additional copies and have them in electronic 
format as well. When the project ended, we removed the separate project webpage and added 
the materials to the PAR^A Center website, so the worksheets and other materials are there 
for the use of people who need them. 

Activity 2.1.4. Collect implementation ideas to include in an on-line “handbook” 

We didn’t do this. Again, when we reached the point where we had enough information to 
publish on the website, we found that each situation was so unique that solutions were rarely 
transferable from one context to the next. Therefore, we elected to provide individual 
assistance rather than trying to write a handbook. 

Objective 2.2. Provide Annual Networking Meetings 

Activity 2.2.1. Establish dates and location for aimual networking meeting 

Activity 2.2.2. Arrange and hold a networking meeting 
We have not held networking meetings as originally planned. However, we have caused 
networking to occur by cormecting people who had similar problems of practice and assisting 
them in collaboratively seeking solutions. 

Activity 2.2.3. Disseminate information about PSA at conferences 
We have disseminated information about PSA at a multitude of conferences. 

Selected Presentations Regarding PSA Outreach at Conferences 
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Principles for Establishing High Quality Training Programs for Paraeducators: Beyond The 
One-Shot Workshop! Paper presented at the National Resource Center for Paraprofessionals, May 1- 
3, 2003, Los Angeles, CA. 

The PAP^A Center Models. Paraprofessional Special Interest Group Meeting, National 
Association of Bilingual Education Conference, January 29 - February 1, 2003. New Orleans. 

The Best Kept Secret: What Teachers Should Know (but don ’t) About Paraeducators: Are we 
preparing teachers for the real world? February 24, 2002, AACTE Conference, New York 

Preparing school professionals to supervise paraprofessionals: One district ’s journey. 
November 16, 2001. 24'*' Arniual Conference of the Council for Exceptional Children Teacher 
Education Division, St. Petersburg. FL. 

Preparing school professionals to supervise paraprofessionals: One district ’s journey. July 
22, 2001, lASA Conference, Warsaw, Poland 

Training Special Education and ESL/Bilingual Paraprofessional, paper presented at the NRC 
Annual Convention, May 12, 2001, Madison, WI 

Training Special Education and ESL/Bilingual Paraprofessionals, paper presented at the CEC 
Annual Convention, April 20, 2001, Kansas City, MO 

Preparation and Roles of Paraprofessional Personnel and the Professionals Who Supervise 
Them. An Invited presentation at the Office of Special Education Programs Project Director’s 
Meeting, February, 22, 2001 Washington, DC. 

Preparing Teachers to Supervise Paraprofessionals in Special Education: Implications for TED. 
Panel presentation with Anna Lou Pickett, Teacher Education Division Conference, Las Vegas, NV, 
November, 2000. 

Title I Paraprofessionals: The Controversy. 31** Annual Title I Parent Involvement Conference. 
October, 2000, Denver, CO 

Preparing School Professionals to Supervise Paraprofessionals. Eighth Annual CSPD Conference on 
Leadership and Change, May, 2000, Alexandria, VA 

Moreover, the findings of the PSA have been published in a variety of forms. This project 
made a major contribution to the content of several chapters and a book written by the PI also 
listed below. 



Edited publications 



French, N. K. (2000). Preparing school professional to supervise paraprofessionals. In K. Murray 
(Ed.), S*** Annual CSPD Conference on Leadership and Change Monograph , (pp. 145-150). Arlington, 



O 

ERIC 

MfliiTIliiniffTlTLiU 



13 



PSA Outreach Final Report 



.Page 13 



VA: Office of Special Education Programs and National Association of State Directors of Special 
Education. 

French, N. K. (1999). Paraeducator Supervision Academy: An Outreach Project to Prepare 
School Professionals to Supervise Paraeducators. In K. Murray (Ed.), 7*^ Annual CSPD Conference 
on Leadership and Change Monograph , (pp. 60-63). Arlington, VA: Office of Special Education 
Programs and National Association of State Directors of Special Educati 

Book and Chapter citations 

French, N. K. (2003). Manasins paraeducators in your school: How to hire, train, and 
supervise classified staff. Thousand Oaks, CA: Corwin Press. 

French, N. K. (2003). Maximizing the Services of Paraeducators. InJ. Burnette (Ed.). Thrivim 
as a Special Educator , (pp. 31-40). Arlington, VA: CEC Publications. 

French, N. K. (2003). Management of paraeducators. InA.L. Pickett andK. Gerlach (Eds.) 
Working with paraeducators in special education: A team approach. 2"‘‘ Edition, (pp. 97-171). 

Austin, TX: Pro-ed, Inc. 

REPORT OF PROJECT EVALUATION GOALS AND OUTCOMES 
We collected data using three different collection methods from multiple sources to 
address evaluation goals 1 -3 and to answer the corollary questions assumed in the three 
evaluation goals. The three goals and corollary questions were: 

Evaluation Goal 1 : Determine the effectiveness of demonstrations as a dissemination device. 
Question 1 : How effective were the demonstrations? 

Question 2: Did the participants perceive that they had gained knowledge and skills as the 
result of the demonstration? 

Evaluation Goal 2: Determine the quality of the follow up support provided by the project 
Question 1 : To what extent were the participants satisfied with the quality and amount of 
support provided by the project? 

Evaluation Goal 3: Evaluate the satisfaction of the participants regarding the quality of the 
Trainers Package. 

Question 1 : To what extent were participants satisfied with the quality of the 
demonstrations and materials? 

Data Collection / Instruments 
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We collected data in three different ways, using two different instruments. Each data 
collection method addressed multiple questions. 

Pre-Post Self-Assessment of Skills 

First, we used questionnaires with 47 items that addressed the self-perceived skills of 
participants regarding each of the topics presented during training. Initially we administered 
the questionnaire prior to and then subsequent to the demonstration / training. Thus we asked 
about participants’ perceptions of their knowledge and skills prior to and subsequent to 
training only after they had completed the training. 

However, we changed the form we used to collect information only at the end of the 
2-day session. We did so for numerous reasons. First, the presenter traveled alone to the 
demonstrate sites without a clerical person or assistant to help vvdth managing material 
distribution, data collection, and other logistics. Although the host sometimes helped vvdth 
these functions, we could not rely on this source of assistance. It was difficult for the 
presenter to remember to distribute forms at two different times. 

Second, some participants came late or left early, making it difficult for the presenter 
to remember to collect forms from those stragglers. Third, because of the various entry and 
exit times of participants, pre-test and post-test forms could not always be matched one-to- 
one. Thus, we quickly realized that our intent of comparing the pre-test score to the post-test 
score of every individual was not realistic under these circumstances. 

Third, we noted some problems in the first sets of data. When we ran initial 
descriptive statistics (including ‘crosstabs’) on the first sets of data we had many empty cells 
because of missing data and there were some cases where the participant actually reported a 
lower self-rating on a topic after instruction had occurred. Moreover, discrepancies existed 
between the verbal feedback to the trainer at the end of the demonstration, email 
conversations subsequent to the demonstrations, the evaluation forms completed by 
participants, all of which tended to be very complimentary and positive, and these pre and 
post self-assessments that showed their skills to be slightly lower subsequent to the training. 

It seemed difficult to believe that participants’ knowledge and skills had actually diminished 
in the course of two days of study on the topic. 
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As we began to explore the reasons for these findings, we found that others who were 
trying to evaluate outcomes of professional development activities had encountered a similar 
phenomenon and had named it “Response-shift bias.” Response-shift bias named the human 
tendency to revise the internal standard forjudging their knowledge and skills or to 
overestimate their knowledge and skills on the pretest and then to change their frame of 
reference during training because of the knowledge they gained about the subject during 
training. 

Retrospective pretests are one way to control for response-shift bias (change in 
perception of functioning after participation in group) and they are more in agreement with 
observed behaviors and other means of behavioral measurement (Howard, 1980; Pohl, 1982; 
Pratt, McGuigan & Katzev, 2000). Thus we began collecting all information about 
participants’ perceptions of their own skills subsequent to the training. 

Levels of Use Interview Protocol 

We also conducted Levels of Use interviews with selected participants to identify 
their use of the concepts and materials, their satisfaction with the quality of the materials, and 
their understanding of the concepts and the materials after participating in a demonstration. 
The Levels of Use interview protocol was developed as part of the Concems-Based Adoption 
Model (CBAM), a model developed in the 1970's to assess, facilitate, and evaluate the 
change process. This model was integral to the design of the project, so the use of the 
instruments was appropriate. 

The CBAM model includes three diagnostic dimensions: (a) Stages of Concern 
(SoC), (b) Levels of Use (LoU), and (c) Iimovation Configuration Map (ICM). The Levels of 
Use (LoU) interview and Innovation Configuration Map (ICM) were the CBAM instruments 
used in this evaluation. The Levels of Use interview assesses where participants are in their 
current use of concepts and materials and the integration of these concepts and tools into 
their programs. The ICM specifies the change agent’s intended use of the materials, concepts, 
and activities. The LoU interview data have been shown to have a .98 correlation with 
observational data in other studies of innovation adoption (Hall & Hord, 1987). 

There are eight, non-sequential, levels of use: 
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Level 0 — Non-Use 

The participant has little or no knowledge of the innovation, has no involvement with the 
innovation, and is doing nothing to become involved. 

Level I— Orientation 

The participant is acquiring information about the innovation, exploring its value, and 
exploring its demands. 

Level II- Preparation 

A date for implementation has been set, and the participant is preparing for use. 

Level III - Mechanical Use 

The participant focuses on the short-term, day-to-day use of the innovation. Changes in 
use are made to meet the participant’s own needs rather than the needs of students. The 
focus is on a step-by-step implementation and often results in a disjointed and superficial 
use. 

Level LVA — Routine 

The participant’s use of the innovation is stabilized, few if any changes are being made, 
and little preparation is given to improving use of the innovation or its impact. 

Level IVB — Refinement 

The participant varies the use of the innovation to increase impact on students. Variations 
are based on knowledge of both short and long-term consequences for the student. The 
focus is on the student. 

Level V - Integration 

The participant combines own efforts in using the innovation with that of colleagues. In 
teaming with other colleagues the participant is working to achieve a collective impact on 
the student. The focus is on the student. 

Level VI- Renewal 

The participant is reevaluating the quality of the innovation and is seeking major 
modifications or alternatives to the present innovation. The participant is examining new 
developments in the field and exploring new goals for self and the system. The focus is 
on the student. 

The appropriate or desired LoU level is dependent on the innovation, the goals of the 
project, and the amount of time that has passed since the introduction of the innovation. In 
this project, the goal is to have as many of the IHE, administrative, and staff developer 
participants as possible routinely engaged in training teachers to effectively direct the work 
of paraeducators. The interviews were conducted well after the demonstrations. For some 
participants, it was as little as one year. For others, it was as much as two years. In the 
meanwhile, many email and phone conversations took place between project staff and 
participants. It was reasonable to expect that at least some of the participants would be able 
to reach the level of routine use in the intervening time period. 
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Because of time and logistical constraint (sites throughout the United States), face-to-face 
interviews, typical for Levels of Use interviews (Hall & Hord, 1987) were not possible. We 
first tried to gain response via email questionnaires, but received so few responses that we 
gave up on that approach. We decided that phone interviews were the most effective way to 
gather the information. We conducted the phone interviews using the questions from the 
Levels of Use protocol. 

Third, we kept all unsolicited email and phone messages from PSA demonstration 
participants. This type of data collection was passive rather than active, in that we merely 
waited irntil an imsolicited message arrived and then added it to the collection. 

Researcher Roles 

Pre-Post Perceived Skills Self-Analyses 

The presenter informed participants that we were collecting information regarding 
their perceived skill acquisition and requested their assistance in completing pre and post 
self-assessment forms. We explained that the information would be used to report perceived 
skills change to the funding agency as part of the project evaluation. The presenter generally 
remained in the room while the forms were being completed, but studiously avoided looking 
at the forms as participants completed them or as participants added their completed forms to 
the stack of forms on the table. Thus the presenter was irnable to identify the participant that 
completed a particular form and thus was unable to associated particular answers with 
particular participants. 

Email and Phone Conversations 

LoU Interviews 

Three interviewers conducted Levels of Use interviews. Each had been previously 
trained to conduct and evaluate Levels of Use interviews. One interviewer is the Project 
Director and is responsible for creating and implementing the project. She was the presenter 
for most of the Paraeducator Supervision Academy demonstrations and arranged most of the 
demonstration logistics with the host. Therefore, she knew the host and some of the 
participants well by the time each demonstration was complete. This introduced some 
possible bias into the interviews. We attempted to limit the influence of bias in the cases 
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where the project director knew the interviewee well by having one of the other two 
interviewers conduct those interviews. 

The second interviewer worked with this project as training coordinator for the final 
three years of the total project period (with the time extension). She arranged some of the 
demonstrations, collected the state information for the customizing of materials to state laws 
and regulations and presented two of the demonstrations. The potential bias certainly 
diminished, but in some ways was similar to that of the project director. To limit the effects 
of bias, we assigned interviews from the demonstrations she conducted to another 
interviewer. 

The third interviewer, although hired at the PAR^A Center after the completion of the 
project, was trained to conduct LoU interviews at the same as the first two interviewers, 
however. She knew none of the demonstration hosts or participants. Therefore, whenever we 
wanted to limit the introduction of potential bias by the first two interviewers, we would 
assign an interview to her. 

The three interviewers conducted the phone interviews with selected participants to 
determine how they had implemented the materials, concepts and activities from the PSA 
training. During the interview each participant was asked if they agreed to participate in the 
interview and upon receiving a positive response, asked if they agreed to have the interview 
taped. Once they had agreed to both of these conditions, the tape was turned on and they 
were asked to identify themselves, their position and place of employment, and then to agree 
again to being taped. The interviewers then preceded asking questions from the protocol. 
Upon completion of the interview, participants were thanked for their time and participation. 
No other rewards for interview participation were offered. 

LoU Interview Participant Selection. LoU Interviewees were selected from master 
lists of participants subsequent to the completion of all demonstrations. We first visually 
scanned the master lists by position or job title selecting all those who were likely to have a 
position in which it was possible that they could use the materials to train others. The 
position titles we selected included ‘professor,’ ‘college faculty,’ ‘staff developer, staff 
development specialist, or supervisor of staff development,’ ‘trainer,’ ‘program coordinator,’ 
‘grant or project coordinator,’ ‘administrator,’ ‘program director,’ ‘training coordinator or 
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director,’ ‘consultant,’ and ‘program specialist.’ We did not select teachers or related services 
providers who may be using materials in their daily work, unless they indicated in some way 
that their position included some training responsibilities. This process resulted in 94 
individuals. We then eliminated 1 8 of those who did not provide an email address. We then 
sent email messages requesting an appointment during identified time blocks in which we 
could conduct an LoU interview over the phone. The content of the message is listed below: 
Hello, 

You participated in a workshop regarding paraeducator supervision some time 
ago. When we provided the workshop, we left copies of our training materials with 
you and expressively gave our permission to use them to carry out more training 
with teachers. We ’re now finishing up the federally-funded project that sponsored 
that workshop and writing our final report. We need to speak with people who 
have used our materials and with those who have not. We've selected you as a 
possible interview participant, and would like to interview you over the phone 
about the usefulness of the program and materials. The interview is brief and to 
the point - it will only take about 15 minutes. We would like to conduct all the 
interviews between Friday, August 16 and Friday, September 13, if possible. If you 
will name a convenient date and time within that timeframe, and provide the phone 
number you want us to use to call you at that time, 1 will arrange to have an 
interviewer call you. 

Thank You, 

Nancy French 

We immediately received 30 messages that bounced back and were undeliverable as 
addressed and one auto-reply indicating that the individual had retired, but gave no other 
contact information. We did a little detective work trying to determine whether the 
undeliverable messages were due to typos or other simple mistakes. After several attempts, if 
we could not successfully deliver the message, we gave up. 

In return, we received five messages saying ‘I have not used the materials yet,’ and 
indicating unwillingness to participate in interviews. However, the majority simply failed to 
reply. We resent the message again five days later, hoping a second reminder would 
encourage more people to reply. 

Interviewees. We scheduled interviews with the 22 volunteers, assuming that their 
knowledge, and insights and their willingness to talk about it, would yield the data needed for 
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our purposes as suggested by Krathwohl (1998). Each interview lasted about twenty 
minutes. Table 2 indicates the numbers and position titles of the LoU interview participants. 
Table 2 

Numbers and Titles of LoU Interviewees 



Number 


Title 


1 


Title I Director 


1 


Special Education Director 


1 


Special Education Coordinator 


8 


Professor / Assistant Professor 


1 


State Coordinator of Special Education and CSPD 
(Comprehensive System of Personnel Development) 


1 


English as a Second Language Coordinator 


1 


Staff Developer 


2 


Director of Pupil Services 


1 


Service Coordinator 


1 


IHE grant Coordinator 


1 


Birth - 3 Program Director 


2 


Consultants 


1 


Special Education Teacher 



Assurances, Confidentiality 

All participants were appraised that one of the purposes of the demonstrations was to 
determine whether they were able to use the materials, concepts, and activities in their daily 
work. We explained that we would be collecting information that would help us document 
their use. We assured participants verbally that their responses to pre and post self-analyses 
of skills and to subsequent interviews would be held confidential in the sense that their real 
identities would not be revealed in any publication or transmission of the data. We also 
informed them that email messages and our notes from phone conversations would also be 
used as data to determine the answers to evaluation questions. We indicated that their 
identities would also be protected through the use of pseudonyms. We jokingly offered to use 
whatever pseudonym they might choose if we reported their responses directly. No one 
offered a pseudonym for our use. 
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For the LoU telephone interviews, the interviewers introduced themselves, explained the 
reason for the interview, and asked interviewees for permission to record the interview on 
tape. The interviewers explained that only aggregated data would be used for the grant 
report. The interviewers assured all of the interviewees that we would not use their names or 
any identifying information in the grant report or in any subsequent publications. All 
interviewees agreed to participate in the process and we then recorded that agreement as part 
of the taped interview. 

Data Analysis 

We conducted three types of data analysis, each consistent with the type of data 
collected and with the questions we wanted to answer in this project evaluation. 

To determine whether participants learned the content, we examined the perceived 
changes in participants’ perceptions of their own knowledge and skills as reported on the pre 
and post self-analyses. Because we changed the forms we used in the middle of the project, 
we conducted two separate examinations of data, separating the cases by type of data 
collection form. 

For each data set, we first conducted both pre and post confirmatory and exploratory 
factor analyses to examine patterns among the responses to individual items and to determine 
whether the response patterns were consistent with the major topical headings and constructs 
shown on the syllabus and discussed diuing the PSA Demonstrations. 

The confirmatory factor analysis used Principal Components Analysis requesting nine 
factors because questions on the pre and post test forms were organized into nine categories 
based on the organi 2 ^tion of demonstration content. Although maximum likelihood factoring 
is considered especially useful for confirmatory factor analysis by some (Tabachnick & 
Fidell, 1989), because it provides a statistical test of significance for goodness-of-fit, it has 
drawbacks when samples are large (Kim & Mueller, 1982). A major problem is the tendency 
to derive more or less factors than may be desirable. 

Findings from the confirmatory analysis were followed with exploratory factoring 
and reliability testing to compare the patterns of participant responses to the underlying 
constructs of the demonstration material. Pre and post exploratory factor analyses were then 
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conducted for each data set and reliability analyses were conducted to test the cohesiveness 
of each of the identified factors. 

We then used the items that loaded on the factors derived from the post exploratory 
analysis to conduct factor by factor t-tests to compare the mean pre and post test scores on 
items that showed a strong relationship to one another. 

Second, to determine whether the concepts, materials and activities were being used, we 
used the LoU rubric (Hall, Loucks, Rutehrford, & Newlove, 1975) in conjunction with an 
Innovation Configuration Map to identify the levels of use and to categorize the reported 
behaviors of participants. We developed an Innovation Configuration Map (ICM) for the 
purposes of data analysis. The ICM describes in rubric form, the observable behaviors that 
define use, acceptable modifications, and non-use of the particular innovation - in this case 
the concepts and materials of the PSA Model. For example, if an interviewee indicated that 
they had offered a class or workshop on paraeducator supervision or had embedded the 
content or materials from PSA into an existing or new course, or if they had given specific 
examples of concepts they have taught, we would consider them a user. An interviewee may 
only have used part of the materials and still be considered a user. An acceptable 
modification of PSA might include modifying the handouts or using similar handouts based 
on the same information, modifying the slides by changing the order, or appearance, or using 
supplemental or more recent articles about paraeducator supervision and training. Examples 
of non-use include not using the handouts, slides, text or articles, providing little or no 
training, giving handouts, materials and articles to teachers in lieu of training. 

Third, we saved all email messages and notes made by project staff during phone 
conversations into a word processing format. Project staff members then reviewed the printed 
data to gain an initial impression of themes or big ideas. We then rechecked the themes by 
listing them and comparing them to the data. We coded the data by linking a single theme to 
each phrase, sentence, or paragraph. As we coded the data, we refined and revised the themes 
until we were certain that the themes satisfactorily reduced the data into meaningful sections. 

FINDINGS 
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The findings are organized in three sections based on the corollary questions: 

1 . Effectiveness of the demonstrations. 

2. Participant satisfaction with the quality and amount of follow up and support. 

3. Participant satisfaction with the quality of demonstrations and materials 

Effectiveness of Demonstrations 

There are two ways of judging the effectiveness of demonstrations. The first is to 
determine whether or not participants perceived that they learned the content. The second is 
to determine whether they are using or sharing the concepts, materials and activities. Three 
types of data contribute to the answer to the effectiveness question: 1) pre and post test data; 
2) LoU interview data; and 3) vmsolicited email and phone messages regarding the quality of 
their experience. 

Pre and post perceived skills self-analysis regarding content learned 

The exploratory factor analysis of the data from the separate pre and post self- 
assessments (Study 1) each yielded six factors. While the specific items from the pre and post 
tests loaded somewhat differently the factors yielded the same factor names. The post-test 
exploratory factors are shown in Table 3 with their reliability coefficients. 



Table 3 

Study 1 Factors and Reliability Coefficients 



Factor 

Number 


Factor Name 


# of Items Loading on 
each Factor 


Reliability 

Coefficient 


1 . 


Training & Skill Performance 


10 


.95 


2. 


Teamwork & Interpersonal Skills 


8 


.94 


3. 


Time Management & Scheduling 


10 


.94 


4 . 


Roles & Responsibilities 


6 


.91 


5. 


Orientation 


7 


.91 


6. 


Legislation & Regulations 


4 


.85 



We then tested the differences between factor means for each of the six factors pre and 
post by determining the means of the scores on the items loading on each factor, the 
subjecting those means to a t-test comparison. 
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Table 4 shows the overall mean, standard deviation, and t-value for study 1 as well as the 
number of paired pre and post test items by factor, the factor means, standard deviations and 
t-values. The differences between the means of the ratings on items on the pre and post tests 
overall and for every factor were significantly different at the .999 level. 

Table 4 



Overall and Factor Means, Standard Deviations, and T-values for Study 1 



Factor 


#of 

pairs 


Mean 


SD 


t-value 


Overall pre 




3.5801 


.973 






526 






-35.31* 


post 




4.9554 


.600 




pre 




2.9296 


1.100 




1 


526 






-39.36* 


post 




4.7062 


.726 




pre 




3.9157 


1.046 




2 


525 






-26.15* 


post 




5.0465 


.635 




pre 




3.9522 


1.155 




3 


455 






-20.63* 


post 




.97314 


.695 




pre 




3.4304 


1.210 




4 


523 






-31.02* 


post 




4.9779 


.706 




pre 




3.1328 


1.130 




5 


523 






-37.19* 


post 




5.0322 


.671 




pre 




4.4226 


1.031 




6 


523 






-19.35* 


post 




5.2416 


.634 





* > .999 



The exploratory factor analysis of the data from the separate pre and post self- 
assessments (Study 2) each yielded five factors. While the specific items from the pre and 
post tests loaded somewhat differently the factors yielded the same factor names. The post- 
test exploratory factors are shovm in Table 5 with their reliability coefficients. 

Table 5 

Study 2 Factors & Reliability Coefficients 



O 
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Factor 

Number 


Factor Name 


# of Items Loading 
on each Factor 


Reliability 

Coefficient 


1 


Training & Skill Performance 


10 


.95 


2 


Teamwork & Interpersonal Skills 


8 


.94 


3 


Time Management & Scheduling 


10 


.94 


4 


Roles & Responsibilities 


6 


.91 


5 


Orientation 


7 


.91 


6 


Legislation & Regulations 


4 


,85 



We then tested the differences between factor means for each of the five factors pre and 
post by determining the means of the scores on the items loading on each factor, the 
subjecting those means to a t-test comparison. 

Table 6 shows the overall mean, standard deviation, and t- value for Study 2 as well as the 
number of paired pre and post test items by factor, the factor means, standard deviations and 
t-values. The differences between the means of the ratings on items on the pre and post tests 
overall and for every factor were significantly different at the .999 level. 




26 



PSA Outreach Final Report 



.Page 26 



Table 6 



Overall and Factor Means. Standard Deviations, and T-values for Study 2 



Factor 


#of 

pairs 


Mean 


SD 


t-value 


Overall pre 




2.9498 


.775 






409 






-41.05* 


post 




4.1651 


.523 




pre 




3.0712 


.939 




1 


399 






-30.18* 


post 




4.1072 


.682 




pre 




3.0681 


.863 




2 


407 






-33.13* 


post 




4.1722 


.573 




pre 




4.1873 


.809 




3 


400 






-33.70* 


post 




.97314 


.522 




pre 




2.5256 


.998 




4 


403 






-35.39* 


post 




4.0273 


.746 




pre 




2.6161 


.923 




5 post 


405 


4.3035 


.591 


-38.56* 


post 




5.2416 


.634 





* > .999 



LoU data regarding effectiveness of demonstration 

The LoU findings are organized in tv^^o sections. The first section contains findings 

related to the seven categories that characterize behaviors at each of the Levels of Use. 

These reported behaviors contribute to the determination of the levels using the scale point 

definitions provided by the LoU rubric. The second section contains data summaries based 

on the scale point definitions of the Levels of Use rubric. 

LoU Seven Categories Findings 
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Knowledge. The Knowledge category (Hall & Hord, 1987) describes the participants 
knowledge about PSA concepts, materials and articles covered in the training. Comments in 
the Knowledge category indicate what the informant knows about the characteristics of PSA 
Outreach, how to use it, and the consequences of its use. 

A special education director commented that this training was “a good outline for 
getting started.” She had incorporated the ideas from this training into the evaluation of 
paraeducators in her district and had used much of the information about roles and 
responsibilities of professionals and paraeducators. A school consultant had used the 
materials and information in an inservice for teachers and paraprofessionals so that they 
could complete work-style analysis, build job descriptions and employ team building skills. 
Using the PSA material, she developed a training manual for newly hired employees with an 
emphasis on responsibilities and delegation, one of the most difficult areas. She also 
developed a list of competencies to use as a checklist in evaluating demonstration of skills. 
She believed that this training gave a structure for continuing training and follow through. 

Five professors and two associate professors have incorporated some of the 
information, concepts and materials delivered in PSA into their courses. One of the 
professors stated that he “focused on activities in the book” used in PSA and used the 
information “to teach preservice teachers how to better supervise paraeducators with an 
emphasis on roles, responsibilities, and ethics.” Another professor included the materials in 
performance based standards tasks for teacher candidates. She incorporated information 
from PSA into her course where teacher candidates participate in hands-on learning, such as 
attending lEP (Individual Education Plan) meetings, to determine how students apply the 
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information in a knowledge - application — action - reflection process of learning. A 
Director of Pupil Services stated that although he was not using the materials personally, 

“two staff members who came with hi m (to PSA) have incorporated the materials into 
training for supervising teachers and aides.” He stated that “materials were useable and 
easily adapted - a springboard for discussion.” He appreciated the expertise and the format. 

Some weaknesses noted in the interviews were that some teachers are not ready for 
all of the information, there was not always time to implement the concepts from the training, 
the supervision concepts were not as detailed as one person would have liked, and one person 
reported that some people did not see paraeducator supervision as their responsibility. 

Acquiring Information. Eight of the informants were looking for more information 
about paraeducator supervision and training. In this category of the LoU rubric (Hall & 

Hord, 1987), data describe how participants solicit information about paraeducator 
supervision and training in a variety of ways, including seeking resource persons, 
corresponding with resource agencies, reviewing books and articles, and making visits. 

One participant had recently attended a seminar on software for paraeducator training. 
Another was looking for more research articles and websites to incorporate in his course for 
preservice teachers. Others were looking for information related to No Child Left Behind 
(2002) and how to meet the new requirements that paraeducators have an associate degree, 
two years of college, or pass a rigorous test might impact supervision and training. Fourteen 
respondents indicated that they had sufficient information for their purposes and were not 
currently looking for more information. 
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Sharing. The Sharing category of the Levels of Use rubric (Hall & Hord, 1987) 
contains information about how participants in the PSA Outreach project discuss the 
innovation with others, sharing ideas, plans, resources, outcomes and problems related to the 
use of PSA materials and information. Thirteen of the participants were sharing inf ormation 
from PSA Outreach with others. A professor was consulting with special education teachers 
and colleagues from other universities. Another professor had met with the national Council 
for Exceptional Children and worked with the IDEA Partnership regarding the roles of the 
teacher in supervision of and delegating to paraprofessionals. Others indicated that they had 
shared materials within their agency, district or building, with principals, school 
psychologists, speech pathologists, etc. A staff development specialist from a state 
department of education had shared the information through a state coalition of trainer of 
trainers. The program coordinator for a college of initial teacher preparation collaboratively 
worked with commimity and state colleges to articulate a four-year degree that included 
some of the concepts and materials from PSA. 

Assessing. The Levels of Use category of Assessing (Hall & Hord, 1987) contains 
participant statements about how they evaluate the use of the innovation or some aspect of it 
informally, by making mental assessments, or formally by collecting and analyzing data. 
Three of the participants reported that they were doing formal assessment of their use of the 
PSA materials. One stated that she “wanted people to have meaningful outcomes from her 
training.” She wanted to make sure that teachers went through the manual she developed 
using PSA information, and that they create written job descriptions for paraprofessionals. 
Another interviewee reported that she always did formal evaluations after training, including 
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training about PSA, and she met individually with participants to discuss how the concepts, 
methods and materials worked. She had sent out a questionnaire at the end of the year to see 
what was helpful and most comments were positive. 

Three participants reported that they conducted informal evaluations. One participant 
asked her colleagues what they liked and did not like about the training she had conducted 
using the materials. The colleagues reported that they liked the paraeducator supervision 
materials the most. The director of a birth-to-three program used the information in a 
training for paraprofessionals and reported that they loved and appreciated it, “Prior to this 
(training), they (paraprofessionals) felt like they were going ‘blind’.” A project coordinator 
at a university reported that “Teachers saw it as a way to open up.” Teachers said they could 
take the worksheets to many places and were able to “transfer what kids need to what you 
(teachers) do.” 

Planning. The Planning category of the Levels of Use Rubric (Hall & Hord, 1987) 
contains statements and information about the short term and long term plans that an 
individual makes during the process of adopting the innovation. One participant reported 
that she was struggling day by day to implement PSA, however, the majority of respondents 
(14) had specific plans for using PSA materials, concepts, or activities. 

A special education coordinator reported that she would conduct additional trainings 
in her district because of staff turnover. Eight of the professors interviewed for this study 
were continuing to use the PSA materials, concepts, and activities, or were expanding their 
use. One professor was going to teach an Internet course, one would have a survey course; 
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another reported that she planned to go back through the materials and use some that she had 
not incorporated yet. 

Two of the eight professors had extensive plans, one to work on a national level to 
develop stronger policies around supervision of paraeducators, and the other had 
incorporated performance tasks that included PSA information, for students in her teacher 
preparation program and she was collecting “feedback on the performance tasks to evaluate 
the proficiency level of teachers.” She stated that she would add information about 
paraeducator supervision and training in the next semester course based on the evaluations. 
She also planned to evaluate how well she taught the information and how well her students 
generalized it to practice as teachers. 

Two participants, a director of pupil services and a staff developer were planning to 
include paraeducators in future trainings. One stated that the materials from PSA Outreach 
would be a beginning point of that training and that she was working with a representative 
from the paraeducator union to get a full class up and running for paraeducators. The 
director of the birth to three program wan planning to make a model using PSA information 
for respite providers, and a special education teacher who was planning to use PSA 
information with admi ni strators stated “training and working with paraeducators is my 
passion.” 

Status Reporting and Performing. Because of the similarity in data collected in each 
of these categories, we combined the information into one section. In describing personal 
stands in relation to the use of PSA materials, concepts and activities and how they carry 
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those out in their own situations, informants ranged from “struggling” and “planning to use” 
to assisting in policy development at a national level. 

A Title I director had used the PSA information when she had been a special 
education director and was planning to use it in the future in her new position. The state 
coordinator of special education and CSPD was not personally using the information, but 
reported that he was in a position to support others in using it. The service coordinator stated 
that she was not quite ready to use PSA information at this time and she did not have a 
system set up to deliver the PSA information. The director of the birth-to-three program 
reported that she was “using concepts of PSA to supervise ‘paras’” however she was not 
training teachers to do so. She added that these materials helped paraeducators to “have a 
clearer idea of what is expected” and gave them more confidence. 

A staff development specialist was implementing much of PSA in a school wide 
program including terminology, expectations, job responsibilities, and definitions of the 
critical role of paraeducators. A special education coordinator reported that she uses 
materials from many sources, including PSA, and has even built a planning period into the 
schedule of paraeducators. An ESL coordinator reported that the training had answered a lot 
of questions she had about paraeducator supervision and training. Both directors of pupil 
services reported that they had used the PSA information in siunmer workshops. A school 
consultant reported that paraeducator supervision and training issues were more clarified and 
systematic as a result of this training, and that she now had a “support system in place for 
information to be heard appropriately and followed through upon.” 
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A professor reported that he was one of only six people in the country that taught a 
course on supervising paraeducators for preservice teachers. Another professor reported that 
PSA information “filled a real need for us on management, and dove-tailed vsdth our work on 
site based inclusive programs.” Still another professor was applying the information to 
teacher training in special education, fi-om the general education perspective, and for teacher 
training in early childhood education. 

Levels of Use Findings 

The participants in this study ranged in their Levels of Use fi'om Non-Use to 
Renewal, with at least one informant falling into each level. Additionally, two participants 
were Past Users - defined as people who had once used the materials, but because of 
changing circumstances were not using them at the time of the interview. Table 7 shows the 
levels of use by titles and positions. 

Table 7 

Levels of Use by Titles and Positions of Interviewee 



Level of Use 




Title / Position 


0 - Non Use 


• 


ESL Coordinator 




• 


Staff Development Specialist/Consultant 




• 


State Coordinator of Special Education / CSPD 




• 


Service Coordinator 




• 


Professor 


I - Orientation 


• 


Director of Birth - 3 Program 




• 


Professor 


II - Preparation 


• 


Director of Pupil Services 


III - Mechanical 


• 


Associate Professor 
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Level of Use 


Title / Position 




• Staff Development Specialist 

• Director of Pupil Services 


IVA - Routine 


• Professor 


IVB - Refinement 


• Professor 

• Consultant 

• Consultant 

• Special Education Coordinator 


V - Integration 


• Teacher 

• Professor 

• Professor 


VI - Renewal 


• Professor 


Past User 


• Grant Project Coordinator 

• Title I Director 



Two participants had used PSA materials and information in the past, but were not 
currently using them. Both thought that they might use them again as a result of this 
interview. One past user had changed positions from special education to Title I and looking 
for information about paraeducator supervision that would relate to the mandates of No Child 
Left Behind (NCLB, 2002), and thought she might use the materials again in her new 
position with paraeducators working in Title I programs. 

Although five interviewees scored in the Non-Use category, one of those, the CSPD 
service coordinator, indicated that although he was not personally using the materials or 
concepts, because of his position, he was supporting others in doing so. The staff 
development specialist was still in the planning stage, trying to integrate everyone’s 
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the materials, but was not using them in a systematic way. Several of the Non-Users 
indicated that as a result of the Levels of Use interview, they would re-look at materials from 
PSA vrith the thought of incorporating them in the future. 

At the Orientation Level of Use, users are acquiring information about the innovation 
and exploring its demands on the user and systems he works vrithin. At the Preparation level, 
the user is preparing for their first use. The three interviewees who were at the Orientation 
and Preparation Levels of Use were using the materials and concepts, but on a somewhat 
limited basis. One had incorporated PSA information into summer workshops and was 
discussing PSA with others in the district. The other was using the information to supervise 
paraeducators, but had not trained other teachers. Both were concerned with the systems, or 
lack of systems, that were in place to support PSA. 

At the Mechanical level, users were focusing their efforts on the day-to-day use of 
PSA. A professor stated that at this point she had made no changes in her use of the PSA 
materials and information, but was working day to day. She hoped to be more organized 
next year. Another person at the Mechanical level of use had worked with elementary 
schools and “was trying to get started in the secondary schools,” but she too was struggling 
with organization. 

The professor who was at the Routine level of use had stabilized the use of PSA 
materials and information vrith no changes. At this level a routine has been established vrith 
little thought given to improving the innovation or its consequences (Hall & Hord, 1987). 

This person stated that she had the “most current, up-to-date, and applicable information and 
would continue to use it in the following year. 
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At the Refinement level of use, the user varies the use of the innovation based on 
formal or informal evaluation to increase the impact on clients. A consultant had surveyed 
her constituents to determine the greatest needs around paraeducator supervision and had 
then provided training to meet those needs. She reported that she had a system in place to 
have the information “heard appropriately and followed through upon.” She had developed a 
manual for teachers and newly hired paraeducators using the materials and information, and 
she had developed a system to be sure that teachers went through the manual and developed 
written job descriptions for paraeducators. A program coordinator at a college of initial 
teacher preparation had included the PSA materials and information in performance-based 
standards for her students. She was currently applying the information to teacher training for 
a special education and general education perspective, and was collecting data and consumer 
feedback to evaluate the proficiency level of teachers in her program in supervising 
paraeducators. Based on the evaluations and data she would incorporate additional pieces of 
PSA in the next semester course. Her concern was “how well we teach it and how well they 
generalize it to practice.” Another participant was using the information from PSA but had 
combined it with other materials that she had received from others. She stated that PSA 
materials “weren’t meeting the needs (of all of her teams) because teams continued to work 
together” and that is why she had added other materials. 

At the Integration level of use, the user is coordinating his efforts with others to have 
a collective impact on clients. Two participants were team teaching a college level class 
about paraeducator supervision using materials and information from PSA. They are also 
using the materials and information in their own classroom. Another participant has been 
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using the materials in her classes at the university, and has also integrated the concepts and 
material into consultation vrith a local community college. 

The Renewal level is the stage where the user explores alternatives or major 
modifications, examines new developments in the field, and explores new goals. This is true 
of one participant who had used the PSA materials and information in the past, but was 
currently focusing on the assessment of appropriate supervision of and planning for 
paraeducators. This professor had been instrumental in requiring a course in supervision for 
a special education endorsement for teachers in his state, and was currently working at a 
national level to implement a course on supervision of paraeducators for preservice teachers. 

Unsolicited email and phone message data 
Content analysis of the emails resulted in four major themes: 

1 . Participants in administrative and trainer roles will share the information with others. 

2. Some professors in colleges of education are incorporating the content into their 
curriculum and courses 

3. Teacher-participants have used the information and have shared it with other teachers 

4. Many participants found the demonstrations of high quality. 

The evidence that participants in administrative and trainer roles are sharing the 
information with others is typified in this statement by a speech language pathologist, who 
holds a training role. 

We have used your materials and find them very valuable...! work with many 
paraeducators as our staff number in the excess of 100 personnel. I have also been 
contracted by the North Dakota DPI to provide inservice in several areas which 
include speech pathology and Autism... most recently three special education units 
have contacted me to provide their paraeducator staff with inservice and I would 
like to include some of your valuable information in my presentation... With my 
current and continued work within the schools for the past 24 years, providing 
personal experience and examples with some of the strategies in management, I 
hope to provide a valuable training. Thank you for our opportunity to hear your 
information and utilize the program within our state. 

Jackie, Speech Language Pathologist MS/CCC (North Dakota) 
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The evidence that colleges and schools of education are incorporating the content 
into their c\irriculum was evident in numerous messages. The first is an excerpt from a 
message sent by the host who helped set up the demonstration in Iowa, and who took it 
upon herself to help disseminate the materials and the concepts to many teacher 
preparation programs in the state. 



Nancy, 

One goal of mine was to write a letter to all the colleges and universities that have 
teacher prep programs in special education and invite them to include your 
materials into their curriculum. As of today, I have 12 out of 22 that have requested 
the materials and plan to use them this year. I told them that I would follow-up to 
learn how they were used and their impression of how beneficial they are to their 
future teachers. Two of the institutions have invited me to come to their class to do a 
presentation, one of the two will actually have me there for 6 hrs and that particular 
university will continue the curriculum with the students who will be doing 
internships the next semester. That's a great beginning, thought you'd like to know. 

4^ Betty, Consultant, Iowa Department of Education 

In response to her message, numerous colleges in Iowa indicated interest and 
willingness to incorporate materials into their curricula. 

Thanks for the ... Paraeducator supervision materials. lam very interested in these 
materials. We will work the concepts into our curriculum. 

^ Bert, St. Ambrose College 

Hello! J..R... here at Mount Mercy College received a letter from you concerning a 
new set of training materials developed for paraeducators. I would love to receive 
these materials as I am the professor in charge of the curriculum for endorsement in 
multi-categorical resource room and we do spend part of at least one class on this 
area. Thank you so much for the information 

^ Ernie, Moimt Mercy College 

At the State University of New York at Stony Brook, there were no full time faculty 
members who were prepared to teach a new course on supervising paraeducators. 
Therefore the faculty members who participated recruited two of the teacher participants 
in the New York demonstration (both master teachers) to develop and teach a course on 
paraeducator supervision as adjimct faculty. Those two teachers were so thrilled with the 
opportunity that they sent weekly messages to project staff as they were developing the 
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course and teaching it the first two times. Several themes emerge from their messages, 
but the most powerful was that the materials provided were a valuable basis on which to 
develop a course. 

...The articles/readings Nancy passed out when she was in NY... have been very 
informative and helpful. The course is being reviewed by Curriculum Committee at 
Stony Brook this month — we should hear soon. Steve and I have been meeting and 
working on lesson plans and the final exam for "THE COURSE. " I have been 
adding suggested readings into the lesson plans. 

Kelly and Steve 

Thanks Nancy for all the material, we are blending it with the Book (Pickett & 
Gerlach, 1997 - a copy of which was given to each site). 

*k Kelly and Steve 

And, on numerous occ^ions they spontaneously offered their thanks. 

We want to thank you all for your support and assistance. This has truly been a 

team effort. We will keep you informed as we go along. 

Sometimes they spoke about their students - teachers to whom the information is being 

disseminated through this Outreach project. 

The students in this class are from all different districts. They are a mixture of 
regular and special education teachers and teachers looking for jobs. ...A great 
group who love to talk and ask questions. 

Often, they were impressed with the learning they witnessed in their students. 

One student stayed after class for 40 minutes just talking with Steve and myself 
about the material presented. She set up a meeting with her principal to go over a 
question sheet she gave her regarding her knowledge and views on paraeducators. 
She is going to used it in her final project. She is not aware how impressed Steve 
and 1 were with all she had to say. On our way out to the cars we looked at each 
other and said they really are learning! This is very exciting!! 

Often they described the materials they had used and how, specifically, they used them. 
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Steve and I added how to work with the OT, PT and speech teacher along with 
incorporating them into the team, when presenting Chapter 3. Nancy, they enjoyed 
the article on Tracy's story. The feeling was that could have been any teacher... 

Steve & I went over paraeducator orientation, Nancy's worksheets and to delegate 
or not to delegate. Some of the worksheets were too advanced for where we (New 
York) are in the process. ...But understandable and workable for the future as the 
process moves forward. 

We ran the video on Together: Building Better Working Relationships from The 
Para Center. 

Steve and I went over teamwork, communication, problem solving 

and conflict resolution. We followed it up with 2 articles from Nancy the CEC one 

& Parent Perspectives. What a way to tie it all together. 

Steve did a great job with the seven executive functions!! I followed up with 
planning using forms and directing and delegating. 

For the Snow Class we gave an assignment— read Nancy's article on Paraeducators 
in Inclusive Classrooms and write a reaction paper. Most of the student are regular 
ed teachers with push in TA 's and some special ed teachers with and without 
paraeducators. The regular ed teachers are learning about special ed. Much 
discussion about inclusion!!! 

At the end of class an Art teacher expressed that at the start of the Class he didn't 
feel that a paraeducator would be appropriate for his class, but after going over the 
material he felt different and can see how a paraeducator it would be beneficial. 
Another teacher asked if special education teachers received classes regarding 
paraeducators? We said no— (except BOCES teachers in trainings) She responded 
with “Not even in college under grad? 

Some email messages spoke specifically about the quality of the demonstration and 
the expertise of the presenter. For example, 

I want to thank you again for bringing your expertise to Maryland. There 
Has been a lot of positive feedback regarding the Academy... 

4k Arlis, Consultant, Maryland State Department of Education 

There was also evidence that teacher participants used the information and shared it 
with others. One teacher wrote to say. 

Thank you so much. I have completed all assignments and I am adding them as 
attachments. Please keep in mind that I have the interesting situation of having the 
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CSC Chairperson ’s paraeducator as my co-teacher in my inclusion class. Our 
replies to duties reflect the ability of my co-teacher to complete almost all tasks in 
the classroom. We do choose to work together each year because of our particular 
flow'. We seem able to complete each other's thoughts in the classroom even 
though we are very different people. 

I really enjoyed the class and it did give me some insight into my relationship with 
my co-teacher / paraeducator. Thanks again, 

4. Maureen (DoDDS, Europe) 

Here is the work I put together for credit from your workshop while with us in 
Vogelweh! I really enjoyed the information and it has already proved valuable! 

4 Pearle, Art Teacher, Kaiserslautern Germany 

Your training extended to other faculty here that did not have the opportunity to 
attend that training. Since my paraprofessional works in their classrooms in the 
inclusion model, I have helped those teachers (via your training) to understand to 
how to have a cooperative and productive relationship with the paraprofessional. 

So, all seems to be going well. 

4 Scott (Yokota Air Force Base, Japan District, DoDDS) 

/ attended the conference you presented in Minnesota regarding paraeducators. 

Our group felt that it was extremely informative ... We are giving a one-day in- 
service to case managers and paraeducators early this fall, and are in the process 
of putting the presentation together. We would very much like to use the 
information from your presentation. Thank you so much for the wealth of well- 
organized and thought out material. Your work is much appreciated. 

4 Barbara (Special Education Department Chair, St. Paul, MN) 

I still have the same paraprofessional that I had when I took your course but am 
losing her in December. I understand her so well now and her "other culture " 
ways. She has become an excellent assistant. Now I will have to learn to 
understand someone new and their new cultural perceptual and behavioral 
differences. It seems that all our paraprofessionals here are from a different 
culture. Thanks for opening my eyes. I have learned to have more patience and not 
see the duty performance through just my own training and personal perceptions. 

4 Scott (Yokota Air Force Base, Japan District, DoDDS) 

CONCLUSIONS AND DISCUSSION 

Our primary conclusion is that project activities were successfully completed and 
they met the intended goals and objectives. The pre and post test self-perceived skills change 
data showed that there were significant differences between the self-perceived skills prior to 
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the PSA demonstration and subsequent to the PSA demonstration. In spite of om initial 
concern that some participants perceived diminished skills subsequent to the demonstration, 
this analysis does not confirm the veracity of that concern. In fact, these data clearly show 
that the perceived skills and knowledge after the demonstration were much greater and match 
both the experience of the presenter, the verbal comments made by participants to the 
presenter, the narrative evaluation comments, as well as the email and phone message 
content. 

The LoU interview data of an imperfect representative sample of participants show 
that about 77% of participants are using the concepts, activities, or materials in some way. 
Although the sample was not satisfactory and the difficulty of reaching past participants 
significant, the question of whether they are using materials is particularly important. The 
emails and phone message data also give specific information about ways in which the 
materials are being used and demonstrate the satisfaction of participants with both the 
materials themselves and with the level of follow up support they received. 

FUTURE PLANS 

We have similar pre-test and post-test data, as well as retrospective pre-then-post test 
data fi"om a very large sample of participants (about 800) who differ substantially in makeup 
than the group of participants in this project. We plan to conduct similar analyses of those 
data and to examine in greater depth the differences in learning among those who hold 
various types positions and to conduct a follow up study with those participants as well. 

We also intend to go back and conduct another series of LoU interviews with a larger 
group of participants from this project to fijrther satisfy the question of whether or not 
participants of this type of staff development program are able to employ the concepts and 
ideas they have learned in whatever capacity they are employed. That question, of comse, 
goes beyond the scope of the completed project but has important applicability in the field. 

Finally, during the demonstrations the presenter conducted an activity at each site that 
asked participants to identify their experiences with paraeducators, then to systematically 
theme those issues in small groups. Each site identified certain issues that transcended 
geographical location. However, many identified issues that were umque to the location, the 
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policies and practices of the school district, college, state legislation or regulations, or just 
common wisdom in the region. The themes were recorded as pent of the activity and we are 
currently conducting an analysis of these data. We believe that these data can be published 
and may have the effect of influencing policies and practices of paraeducator employment, 
training and supervision across the nation. 

SUMMARY 

This project met all its intended goals and has served to disseminate materials to a 
broad audience across the country. It has collected data which may be published to help 
advance the knowledge base in this newly emerging field. It has established a basis upon 
which to continue in this line of research and poses many new questions that beg answers. 
Finally, it creates a foundation for future work in policy and practice in the field. 
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